Gui element acquisition using a plurality of alternative representations of the gui element

ABSTRACT

A method, apparatus and product for GUI element acquisition using a plurality of alternative representations of the GUI element. The method comprising: obtaining a selection of a Graphical User Interface (GUI) element; obtaining a representation set of the GUI element, wherein the representation set comprises a plurality of alternative representations, each of which provides a manner of identifying the GUI element in GUIs; applying the representation set to identify the GUI element in a GUI of a program, wherein said applying the representation set comprises, for each representation of the alternative representations, independently applying the each representation to identify one or more GUI elements corresponding the each representation; and in response to determining that said applying the representation set provides an indefinite result, obtaining a modified representation set.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation under 35 U.S.C. § 120 ofInternational Application PCT/IL2019/050475, filed Apr. 30, 2019, thecontents of which are incorporated by reference herein

TECHNICAL FIELD

The present disclosure relates to Graphical User Interfaces (GUIs) ingeneral, and to GUI element acquisition, in particular.

BACKGROUND

An application having a Graphical User Interface (GUI) may take manyforms. It may have a web version, running in browsers such as CHROME™,FIREFOX™, or the like. Additionally or alternatively, the applicationmay have a desktop version, running on WINDOWS™, OSX™, or the like.Additionally or alternatively, the application may have a native mobileversion, running on iOS™, ANDROID™, or the like. Additionally oralternatively, the application may have a web mobile version, running onmobile browsers. Additionally or alternatively, the application may havebeen written using a cross platform framework such as XAMARIN™, REACTNATIVE™, or the like.

Due to differences in the underlying platform, each version may bedifferent. For example, even using the same programming language mayresult in different outcomes, such as the case due to differentJavaScript engines for each platform. Additionally or alternatively, dueto different screen sizes, capabilities, or the like, the sameapplication may have different GUI on each platform.

Additionally or alternatively, during the life cycle of an applicationit may change. Features may be added, removed, or change place.Additionally or alternatively, the User eXperience (UX) may change,resulting in a changing GUI over time. Additionally or alternatively,users with different roles or permissions may see different views of thesame screen.

BRIEF SUMMARY

One exemplary embodiment of the disclosed subject matter is a methodcomprising: obtaining a selection of a Graphical User Interface (GUI)element; obtaining a representation set of the GUI element, wherein therepresentation set comprises a plurality of alternative representations,each of which provides a manner of identifying the GUI element in GUIs;applying the representation set to identify the GUI element in a GUI ofa program, wherein said applying the representation set comprises, foreach representation of the alternative representations, independentlyapplying the each representation to identify one or more GUI elementscorresponding the each representation; and in response to determiningthat said applying the representation set provides an indefinite result,obtaining a modified representation set.

Optionally, the alternative representations comprise a representation,wherein said applying the representation set provides the indefiniteresult due to failing to identify any element in the GUI using therepresentation.

The method of claim 1, wherein the alternative representations comprisea first alternative representation and a second alternativerepresentation; wherein said applying the representation set providesthe indefinite result due to identifying a first element using the firstrepresentation and identifying a second element using the secondrepresentation, wherein the first element and the second element aredifferent.

Optionally, said applying the representation set provides the indefiniteresult due to identifying two or more elements using an alternativerepresentation.

Optionally, said applying the representation set provides the indefiniteresult due to an alternative representation of the alternativerepresentations, wherein said obtaining the modified representation setcomprises removing the alternative representation from therepresentation set, whereby applying the modified representation set inthe GUI of the program provides a definite result.

Optionally, the indefinite result was caused by a first representation,wherein the first representation is comprised by the plurality ofalternative representations, wherein said obtaining the modifiedrepresentation set comprises removing the first representation from therepresentation set and adding a new representation to the representationset, wherein the new representation is configured to uniquely identifythe GUI element in the GUI of the program, whereby applying the modifiedrepresentation set on a GUI of the program provides a definite result.

Optionally, the method further comprises: applying the newrepresentation on an original GUI, wherein the representation set wasdetermined based on the original GUI, whereby the new representation isalso configured to uniquely identify the GUI element in the originalGUI.

Optionally, said applying the representation set is performed on aclient device; wherein in response to the client device determining thatsaid applying the representation set provides the indefinite result,communicating information to a server, wherein the server is configuredto modify the representation set to determine the modifiedrepresentation set; and wherein said obtaining the modifiedrepresentation set comprises obtaining by the client device the modifiedrepresentation set from the server.

Optionally, a server is configured to define the representation set,wherein the server is configured to transmit the representation set to aplurality of client devices; wherein said applying the representationset is performed on a client device; wherein in response to the clientdevice determining that said applying the representation set providesthe indefinite result, communicating information to the server, whereinthe server is configured to modify the representation set to determinethe modified representation set and to transmit the modifiedrepresentation set to the plurality of client devices; and wherein saidobtaining the modified representation set comprises obtaining by adevice the modified representation set from the server, wherein thedevice is comprised by the plurality of client devices.

Optionally, said applying the representation set is performed aplurality of times, each of which with respect to a different GUI,wherein the different GUIs are retrieved from a historic database,retaining GUIs previously encountered by computerized agents.

Optionally, an alternative representation uses at least one of: one ormore attributes of the GUI element; one or more attributes of anotherGUI element; a relative path to the GUI element in a hierarchy ofelements; and an absolute path to the GUI element in the hierarchy ofelements.

Optionally, said obtaining the representation set comprises: determininga set of alternative representations; and selecting a subset from theset of alternative representations, whereby selecting the plurality ofalternative representations; and wherein the modified representation setis obtained by selecting a different subset from the set of alternativerepresentations.

Optionally, the method further comprises, in response to determiningthat said applying the representation set provides an indefinite result:displaying the GUI to a user; identifying the GUI element in the GUIbased on user input from the user; and wherein said obtaining themodified representation set comprises determining a new set ofalternative representations that uniquely identify the GUI element inthe GUI, wherein the modified representation set comprises the new setof alternative representations.

Optionally, said displaying the GUI comprises displaying the GUI andmarking a suggested element, wherein the user input is a confirmation ofthe suggested element.

Optionally, said obtaining the representation set comprises applying apredictive model to select the plurality of alternative representationsfor the representation set from a set of alternative representations;wherein the predictive model is trained based on GUI elements andalternative representations thereof.

Optionally, said obtaining the modified representation set comprisesapplying a predictive model to select an alternative representation tobe removed from the representation set, wherein the predictive model istrained based on data that comprises manual modifications torepresentation sets.

Optionally, the representation set comprises the plurality ofalternative representations and one or more additional alternativerepresentations; wherein the method further comprises: obtaining acurrent GUI context of the GUI; and wherein said applying therepresentation set comprises selecting the plurality of alternativerepresentations based on the GUI context and applying only the pluralityof alternative representations without applying the one or moreadditional alternative representations.

Optionally, the GUI element is selected from a group consisting of acheckbox, a click button, a radio button, a dropdown list, a dropdownbutton, a toggle button, a text field, a date and time picker, a searchfield, a breadcrumb, a pagination, a tag, a slider, an icon, an imagecarousel, a notification, a progress bar, a tool tip, a message box, amodal window, a widget, a menu bar, a tab, a table, an image element, alabel, and an accordion.

Optionally, the representation set comprises a set of alternativerepresentations, wherein the set of alternative representations comprisethe plurality of alternative representations and at least one morealternative representation; and wherein said applying the representationset comprises determining a subset of the set of alternativerepresentations to be applied on the GUI, wherein the subset consists ofthe plurality of alternative representations, whereby said applyingselectively applies only a portion of available alternativerepresentations.

Another exemplary embodiment of the disclosed subject matter is acomputer program product comprising a non-transitory computer readablestorage medium retaining program instructions, which programinstructions when read by a processor, cause the processor to perform:obtaining a selection of a Graphical User Interface (GUI) element;obtaining a representation set of the GUI element, wherein therepresentation set comprises a plurality of alternative representations,each of which provides a manner of identifying the GUI element in GUIs;applying the representation set to identify the GUI element in a GUI ofa program, wherein said applying the representation set comprises, foreach representation of the alternative representations, independentlyapplying the each representation to identify one or more GUI elementscorresponding the each representation; and in response to determiningthat said applying the representation set provides an indefinite result,obtaining a modified representation set.

Optionally, the alternative representations comprise a representation,wherein said applying the representation set provides the indefiniteresult due to failing to identify any element in the GUI using therepresentation.

Optionally, the alternative representations comprise a first alternativerepresentation and a second alternative representation; wherein saidapplying the representation set provides the indefinite result due toidentifying a first element using the first representation andidentifying a second element using the second representation, whereinthe first element and the second element are different.

Optionally, said applying the representation set provides the indefiniteresult due to identifying two or more elements using an alternativerepresentation.

Optionally, said applying the representation set provides the indefiniteresult due to an alternative representation of the alternativerepresentations, wherein said obtaining the modified representation setcomprises removing the alternative representation from therepresentation set, whereby applying the modified representation set inthe GUI of the program provides a definite result.

Optionally, the indefinite result was caused by a first representation,wherein the first representation is comprised by the plurality ofalternative representations, wherein said obtaining the modifiedrepresentation set comprises removing the first representation from therepresentation set and adding a new representation to the representationset, wherein the new representation is configured to uniquely identifythe GUI element in the GUI of the program, whereby applying the modifiedrepresentation set on a GUI of the program provides a definite result.

Optionally, said program instructions when read by the processor,further cause the processor to perform: applying the new representationon an original GUI, wherein the representation set was determined basedon the original GUI, whereby the new representation is also configuredto uniquely identify the GUI element in the original GUI.

Optionally, said applying the representation set is performed on aclient device; wherein in response to the client device determining thatsaid applying the representation set provides the indefinite result,communicating information to a server, wherein the server is configuredto modify the representation set to determine the modifiedrepresentation set; and wherein said obtaining the modifiedrepresentation set comprises obtaining by the client device the modifiedrepresentation set from the server.

Optionally, a server is configured to define the representation set,wherein the server is configured to transmit the representation set to aplurality of client devices; wherein said applying the representationset is performed on a client device; wherein in response to the clientdevice determining that said applying the representation set providesthe indefinite result, communicating information to the server, whereinthe server is configured to modify the representation set to determinethe modified representation set and to transmit the modifiedrepresentation set to the plurality of client devices; and wherein saidobtaining the modified representation set comprises obtaining by adevice the modified representation set from the server, wherein thedevice is comprised by the plurality of client devices.

Optionally, said applying the representation set is performed aplurality of times, each of which with respect to a different GUI,wherein the different GUIs are retrieved from a historic database,retaining GUIs previously encountered by computerized agents.

Optionally, said obtaining the representation set comprises: determininga set of alternative representations; and selecting a subset from theset of alternative representations, whereby selecting the plurality ofalternative representations; and wherein the modified representation setis obtained by selecting a different subset from the set of alternativerepresentations.

Optionally, said program instructions when read by the processor,further cause the processor to perform, in response to determining thatsaid applying the representation set provides an indefinite result:displaying the GUI to a user; identifying the GUI element in the GUIbased on user input from the user; and wherein said obtaining themodified representation set comprises determining a new set ofalternative representations that uniquely identify the GUI element inthe GUI, wherein the modified representation set comprises the new setof alternative representations.

Optionally, said displaying the GUI comprises displaying the GUI andmarking a suggested element, wherein the user input is a confirmation ofthe suggested element.

Optionally, said obtaining the representation set comprises applying apredictive model to select the plurality of alternative representationsfor the representation set from a set of alternative representations;wherein the predictive model is trained based on GUI elements andalternative representations thereof.

Optionally, said obtaining the modified representation set comprisesapplying a predictive model to select an alternative representation tobe removed from the representation set, wherein the predictive model istrained based on data that comprises manual modifications torepresentation sets.

Optionally, the representation set comprises the plurality ofalternative representations and one or more additional alternativerepresentations; wherein said program instructions when read by theprocessor, further cause the processor to perform: obtaining a currentGUI context of the GUI; and wherein said applying the representation setcomprises selecting the plurality of alternative representations basedon the GUI context and applying only the plurality of alternativerepresentations without applying the one or more additional alternativerepresentations.

Optionally, the representation set comprises a set of alternativerepresentations, wherein the set of alternative representations comprisethe plurality of alternative representations and at least one morealternative representation; and wherein said applying the representationset comprises determining a subset of the set of alternativerepresentations to be applied on the GUI, wherein the subset consists ofthe plurality of alternative representations, whereby said applyingselectively applies only a portion of available alternativerepresentations.

Yet another exemplary embodiment of the disclosed subject matter is anapparatus comprising a processor and a coupled memory, the processorbeing adapted to perform the steps of: obtaining a selection of aGraphical User Interface (GUI) element; obtaining a representation setof the GUI element, wherein the representation set comprises a pluralityof alternative representations, each of which provides a manner ofidentifying the GUI element in GUIs; applying the representation set toidentify the GUI element in a GUI of a program, wherein said applyingthe representation set comprises, for each representation of thealternative representations, independently applying the eachrepresentation to identify one or more GUI elements corresponding theeach representation; and in response to determining that said applyingthe representation set provides an indefinite result, obtaining amodified representation set.

Optionally, the alternative representations comprise a representation,wherein said applying the representation set provides the indefiniteresult due to failing to identify any element in the GUI using therepresentation.

Optionally, the alternative representations comprise a first alternativerepresentation and a second alternative representation; wherein saidapplying the representation set provides the indefinite result due toidentifying a first element using the first representation andidentifying a second element using the second representation, whereinthe first element and the second element are different.

Optionally, said applying the representation set provides the indefiniteresult due to identifying two or more elements using an alternativerepresentation.

Optionally, said applying the representation set provides the indefiniteresult due to an alternative representation of the alternativerepresentations, wherein said obtaining the modified representation setcomprises removing the alternative representation from therepresentation set, whereby applying the modified representation set inthe GUI of the program provides a definite result.

Optionally, the indefinite result was caused by a first representation,wherein the first representation is comprised by the plurality ofalternative representations, wherein said obtaining the modifiedrepresentation set comprises removing the first representation from therepresentation set and adding a new representation to the representationset, wherein the new representation is configured to uniquely identifythe GUI element in the GUI of the program, whereby applying the modifiedrepresentation set on a GUI of the program provides a definite result.

Optionally, said processor is further configured to perform: applyingthe new representation on an original GUI, wherein the representationset was determined based on the original GUI, whereby the newrepresentation is also configured to uniquely identify the GUI elementin the original GUI.

Optionally, said applying the representation set is performed on aclient device;

wherein in response to the client device determining that said applyingthe representation set provides the indefinite result, communicatinginformation to a server, wherein the server is configured to modify therepresentation set to determine the modified representation set; andwherein said obtaining the modified representation set comprisesobtaining by the client device the modified representation set from theserver.

Optionally, the apparatus is a client device, wherein a server isconfigured to define the representation set, wherein the server isconfigured to transmit the representation set to a plurality of clientdevices; wherein said applying the representation set is performed bysaid client device; wherein in response to the client device determiningthat said applying the representation set provides the indefiniteresult, communicating information to the server, wherein the server isconfigured to modify the representation set to determine the modifiedrepresentation set and to transmit the modified representation set tothe plurality of client devices; and wherein said obtaining the modifiedrepresentation set comprises obtaining by a device the modifiedrepresentation set from the server, wherein the device is comprised bythe plurality of client devices.

Optionally, said applying the representation set is performed aplurality of times, each of which with respect to a different GUI,wherein the different GUIs are retrieved from a historic database,retaining GUIs previously encountered by computerized agents.

Optionally, said obtaining the representation set comprises: determininga set of alternative representations; and selecting a subset from theset of alternative representations, whereby selecting the plurality ofalternative representations; and wherein the modified representation setis obtained by selecting a different subset from the set of alternativerepresentations.

Optionally, said processor is further configured to perform, in responseto determining that said applying the representation set provides anindefinite result: displaying the GUI to a user; identifying the GUIelement in the GUI based on user input from the user; and wherein saidobtaining the modified representation set comprises determining a newset of alternative representations that uniquely identify the GUIelement in the GUI, wherein the modified representation set comprisesthe new set of alternative representations.

Optionally, said displaying the GUI comprises displaying the GUI andmarking a suggested element, wherein the user input is a confirmation ofthe suggested element.

Optionally, said obtaining the representation set comprises applying apredictive model to select the plurality of alternative representationsfor the representation set from a set of alternative representations;wherein the predictive model is trained based on GUI elements andalternative representations thereof.

Optionally, said obtaining the modified representation set comprisesapplying a predictive model to select an alternative representation tobe removed from the representation set, wherein the predictive model istrained based on data that comprises manual modifications torepresentation sets.

Optionally, the representation set comprises the plurality ofalternative representations and one or more additional alternativerepresentations; wherein said processor is further configured toperform: obtaining a current GUI context of the GUI; and wherein saidapplying the representation set comprises selecting the plurality ofalternative representations based on the GUI context and applying onlythe plurality of alternative representations without applying the one ormore additional alternative representations.

Optionally, the representation set comprises a set of alternativerepresentations, wherein the set of alternative representations comprisethe plurality of alternative representations and at least one morealternative representation; and wherein said applying the representationset comprises determining a subset of the set of alternativerepresentations to be applied on the GUI, wherein the subset consists ofthe plurality of alternative representations, whereby said applyingselectively applies only a portion of available alternativerepresentations.

THE BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosed subject matter will be understood and appreciatedmore fully from the following detailed description taken in conjunctionwith the drawings in which corresponding or like numerals or charactersindicate corresponding or like components. Unless indicated otherwise,the drawings provide exemplary embodiments or aspects of the disclosureand do not limit the scope of the disclosure. In the drawings:

FIG. 1 shows a schematic illustration of an exemplary environment andarchitecture in which the disclosed subject matter may be utilized, inaccordance with some exemplary embodiments of the disclosed subjectmatter;

FIGS. 2-4 show flowchart diagrams of methods, in accordance with someexemplary embodiments of the disclosed subject matter;

FIGS. 5A-5E shows exemplary screens, in accordance with some exemplaryembodiments of the disclosed subject matter;

FIG. 6A shows a block diagram of an apparatus, in accordance with someexemplary embodiments of the disclosed subject matter; and

FIG. 6B shows a block diagram of an apparatus, in accordance with someexemplary embodiments of the disclosed subject matter.

DETAILED DESCRIPTION

One technical problem dealt with by the disclosed subject matter is torepresent a GUI element, preferably in a robust manner. In someexemplary embodiments, a GUI element may be a button, a label, atooltip, a widget, an input field, a text box, or the like. It may bedesired to automatically acquire a GUI element in a given GUI in orderto allow execution of walkthroughs, gathering information for analytics,process automation, adding a display layer, augmenting functionality ofthe GUI, or the like.

As an example, a walkthrough may be a tutorial which, when executed,presents, in a sequential manner, descriptive elements relating to theGUI, such as disclosed in U.S. Pat. No. 9,922,008, entitled “CallingScripts Based Tutorials”, which is hereby incorporated by reference inits entirety for all purposes without giving rise to disavowment. Thedescriptive elements may include instructions and/or examples of usage,and may be associated with respective GUI elements, such as a specificHTML object within a web document. In order to be able to place thedescriptive element near the respective GUI element, there is a need toautomatically acquire the GUI element. The acquisition of the GUIelement may be based on a representation thereof. If, however, there isa change in the GUI, the representation may not be robust, and theacquisition may fail.

In some exemplary embodiments, a GUI element may not have a uniqueidentifier in the GUI that is consistent over time. As an example, abutton “OK” may not have any ID associated with it in the source code.There may be multiple buttons with the text “OK”. Hence, it may not befeasible to acquire a button by searching for buttons with text “OK”.

Additionally or alternatively, there may be different versions of a GUI,such as a web page, each of which may have different text. Hence, thebutton with text “OK” may be found by searching for the word “OK” in theEnglish version of the web page and may not be found in other versionsof the web page such as the German version, the Hebrew version, theChinese version, or the like.

Additionally or alternatively, the code of the GUI may be dynamicallyobfuscated. For example, a web page may be obfuscated each time it isretrieved. As an example, a GUI element in a web page may have an “ID”property which is “send button” in the original source code. Each timethe web page is retrieved, an obfuscator may choose a new and random IDproperty. Trying to acquire the GUI element based on the ID “sendbutton” may be plausible during testing but may fail in production.Additionally or alternatively, as an obfuscator may provide a new IDeach time the page is retrieved, a successful acquisition may notprovide a hint as to an ID that can be used in future acquisitions.

Another technical problem dealt with by the disclosed subject matter isto robustly represent a GUI element in a program that changes over time.A program may change over time as a result of adding new features,removing features, design changes, bug fixes, or the like. It may bedesired to find a robust representation. Such robust representation maybe needed in order to find the same element over time. As an example, anapplication may have a “Pay Now” button. The application may change overtime. For example, the “Pay Now” button may change place within theapplication, the size of the button may change, the text may change, orthe like. It may be desired to collect usage information automaticallyregarding how many users have pressed the pay button, how many usershave completed the payment, how long did it take for a user to completethe purchase, or the like. Such collected data may be used, for example,in providing useful funnel analysis of the program. Having arepresentation of the payment button may allow to acquire the paymentbutton in the GUI, and identify interactions therewith. A representationthat is not robust over time may fail to allow acquisition of the samebutton, and therefore may miss interactions with the GUI button (andpotentially account interactions with a different GUI element instead).Hence, a representation that is not robust over time may fail to providea reliable automatic usage analysis. Similarly, a non-robustrepresentation may create a technical challenge for automaticperformance of user interactions, such as disclosed, for example, in thesubject matter of U.S. Pat. No. 9,934,782, entitled “Automaticperformance of user interaction operations on a computing device”, whichis hereby incorporated by reference in its entirety for all purposes andwithout giving rise to disavowment.

Yet another technical problem dealt with by the disclosed subject matteris to robustly represent a GUI element in a program that changes overplatforms. As an example, a program may have a native version forWINDOWS™ and a hybrid version for mobile devices. It may be desirable torobustly represent a GUI element in a manner that will allow acquisitionof the GUI element through the different versions.

One technical solution is to obtain a GUI of a program, such as ascreen, a form, a web document, an ANDROID™ layout, an iOS™ playgroundobject, or the like. The GUI of the program may comprise a plurality ofGUI elements, including a GUI element of interest. A representation setof the GUI element may be determined and used as a robust representationthereof. The representation set may comprise a plurality of alternativerepresentations. In some exemplary embodiments, the representation setmay be applied on the screen, where the GUI element is to be acquired.In some exemplary embodiments, the representation set may be applied byindependently applying each of the alternative representations. In someexemplary embodiments, each of the alternative representations may besufficient to uniquely identify the GUI element. The representation setmay be considered as providing a definite result if the results of thealternative representations are in agreement and identify a singleelement in the screen. Additionally or alternatively, in case theapplication of the representation set provided an indefinite result, therepresentation set may be modified. The modified representation set maybe a modification that uniquely identifies the GUI element in thescreen. Additionally or alternatively, the modified representation setmay be useful for acquiring the GUI element in the screen.

In some exemplary embodiments, an alternative representation may use anattribute of the GUI element to identify the GUI element. The attributemay be an identifier of the element, a text associated with the element,the color of the element, or the like. Additionally or alternatively, analternative representation may use an absolute path to the GUI elementin a hierarchy of GUI elements in the screen. In some exemplaryembodiments, a GUI element may comprise one or more sub-GUI elements.GUI elements may be represented in a Directed Acyclic Graph (DAG), atree, or another form of a graph. The GUI may comprise a root element,and a path from the root element to the GUI element may be considered asan absolute path thereto. Additionally or alternatively, an alternativerepresentation may use a relative path from a source GUI element to thetarget GUI element. A relative path to a GUI element may be the pathfrom a source GUI element to the target GUI element. As an example, inwebsites there may be a GUI element with the tag ‘HTML’, which is theroot element, from which all other GUI elements are reachable. As anexample, referring to Screen 500 a illustrated in FIG. 5A. Menu 501 amay be a GUI element which comprises Accounts 502 a, as well as otherelements. A relative path from Menu 501 a to Accounts 502 a may be “the5^(th) son of Menu 501 a”. Such representation may allow to acquire theGUI element Accounts 502 a, based on an acquisition of Menu 501 a.Additionally or alternatively, an absolute path may in Screen 500 a toAccounts 502 a, may be “the 1^(st) item that is a menu→the 5^(th) menuitem”, assuming that Menu 501 a is the first item in Screen 500 a thatis characterized as being a menu. In some exemplary embodiments, thealternative representation may use one or more attributes of one or moreother GUI elements to represent the GUI element. For example, analternative representation may use a query to identify other elements orstructures, and select the GUI element therebetween based on a property,such as structural property, attribute of the element, or the like. Asan example, an alternative representation may be “find an element whichis the deepest root of sub-tree containing elements with attributes <A1>and <A2>”. In this example, all sub-trees containing an element withattribute <A1> and an element with attribute <A2> (which may be the sameelement or a different element), may be identified. Each root of suchsub-trees may be examined, and the node that functions as a root of suchsub-tree and has a largest distance from the root of the treerepresenting the GUI may be selected.

In some exemplary embodiments, a Document Object Model (DOM) may providea tree structure representation for a GUI, such as a GUI based on anHTML, a XHTML, or a XML document. In the logical tree structure of theDOM, each node may be an object representing a part of the document. DOMmay be utilized to traverse a desired path in the GUI.

Additionally or alternatively, the representation set may comprise aplurality of alternative representations. Modification of therepresentation set may comprise removing therefrom one or morealternative representations. Additionally or alternatively, the modifiedrepresentation set may be determined by selecting a subset of theplurality of alternative representations.

In some exemplary embodiments, the modified representation set may bedetermined by removing a first alternative representation from therepresentation set. Additionally or alternatively, the modifiedrepresentation set may be determined by adding a second alternativerepresentation to the representation set, different than the firstalternative representation. In some exemplary embodiments, the secondalternative representation may be characterized in that it uniquelyidentifies the GUI element.

In some exemplary embodiments, applying the modified representation seton the GUI of the program may provide a definite result. Additionally oralternatively, applying the modified representation set may uniquelyidentify the GUI element.

Another technical solution is to apply the representation set in aclient device of an end user. The end user may use a browser running onthe client device, an application running on the client device, or thelike. In some exemplary embodiments, an agent running on the clientdevice may obtain a representation set. Additionally or alternatively,the agent may apply the representation set on screen currently beingwatched by the end user. The agent may be utilizing the representationset in order to acquire the GUI element for performing a predeterminedprocess. The agent may determine that applying the representation setprovided an indefinite result. In response to determining that applyingthe representation set provided an indefinite result, the agent maycommunicate the information to a server. The server may be configured tomodify the representation set as to determine the modifiedrepresentation set. In some exemplary embodiments, the agent may obtainthe modified representation set from the server for future uses. In someexemplary embodiments, the server may distribute, in a push or pullmanner, the modified representation set to a plurality of agents thatmay be configured to acquire the GUI element.

In some exemplary embodiments, the server may store the informationreceived from the agents. The information may be used by a machinelearning algorithm. The machine learning algorithm may be configured toclassify alternative representations in order to match alternativerepresentations with GUI elements.

Another technical solution is to apply the representation set aplurality of times, each of which with respect to a different screen.The different screens may be retrieved from a historic database. Thehistoric database may retain screens previously encountered bycomputerized agents, such as crawlers, scripts, software products, bots,user-operated clients or the like. In some exemplary embodiments, theagent may be a user-operated client, where the user may instruct theclient to save a specific screen encountered by the user in a session.An agent may be executed on a client device and communicate theinformation to a server. The server may store the information in orderto use the information in accordance with the disclosed subject matter.

In some exemplary embodiments, in response to determining that applyingthe representation set provided an indefinite result, the GUI may bedisplayed to a user. In some exemplary embodiments, the user may be anadministrative user, different than the end user using the device inwhich the representation set failed to provide a definite result. Theuser may identify the GUI element in the GUI, select the GUI element inthe GUI, mark the GUI element in the GUI, or the like. In some exemplaryembodiments, in case that the indefinite result comprises more than oneGUI element, the GUI elements identified by the representation set maybe displayed to a user. The user may select one GUI elementtherebetween. Additionally or alternatively, a computerized process mayautomatically select a top candidate out of the more than one GUIelement, and allow the user to verify the automatic selection. Theserver may store the selection of the user and modify the representationset based thereon.

In some exemplary embodiments, selections of users may be recorded andsaved for a future use. As an example, selection of the user may be usedas parts of data for a supervised machine learning algorithm.

In some exemplary embodiments, determining the representation set maycomprise applying an output of a predictive model. The predictive modelmay be the output of a machine learning algorithm configured to receiveas input alternative representations, screens, GUI elements, GUIcontexts, or the like.

One technical effect of utilizing the disclosed subject matter is arobust acquisition of a GUI element. In some exemplary embodiments, theacquisition may be accomplished automatically by an agent. The agent maybe configured to perform a useful process using the acquired element,such as displaying and executing a walkthrough, providing an augmenteddisplay over the GUI, analyzing the actions of the user or users,executing testing automation, or the like. The robust acquisition may beperformed with respect to different versions of the program, differentversion of the GUI, or the like, without requiring manual interventionor assistance.

Another technical effect of utilizing the disclosed subject matter is todetermine a robust representation of a GUI element. The robustrepresentation may comprise a plurality of non-robust representations.In some exemplary embodiment, the disclosed subject matter may be usedin order to choose a representation set comprising a subset ofalternative representation. A GUI of a program may change over time dueto changes in versions, in UX, in functionality of the program, or thelike. Utilizing the disclosed subject matter may enable to acquire a GUIelement in the GUI, even if the GUI had changed from the GUI that wasavailable when the representation was defined. In some exemplaryembodiments, a GUI element may change, a GUI of a program comprising theGUI element may change, or the like. As an example, a GUI element may beacquired by using a text that is associated with the GUI element. TheGUI element may be button comprising the text “Like”. A new version ofthe GUI of a program may change the text to another language, to anicon, or the like. The disclosed subject matter may provide a method toacquire the GUI element regardless of such changes.

Yet another technical effect of utilizing the disclosed subject matteris to enable automation processes of programs, such as execution ofbusiness processes using the GUI without using dedicated ApplicationProgramming Interfaces (APIs), testing automation, or the like. Robustacquisition of GUI elements may enable the automation process tofunction correctly, and thus reduce resources required for re-executionafter manual inspection of error.

Yet another technical effect of utilizing the disclosed subject matteris to enable enhancement of the GUI, such as augmenting functionalitythereof, providing adaptable walkthroughs thereto, adding display layerover the GUI, collecting usage statistics, or the like. The GUIenhancement may require automatic acquisition of GUI elements. Thedisclosed subject matter may provide a robust acquisition, enabling GUIenhancements to function properly over time, in spite of changes to theGUI.

Yet another technical effect of utilizing the disclosed subject mattermay be an improvement of resource consumption in the process ofutilizing a representation of GUI elements. The disclosed subject mattermay provide a robust representation that is durable over time, and avoidthe need to redefine a representation with each modification of the GUI,or for each alternative version of the GUI, such as in case of differentlanguages, different platforms, or the like.

Yet another technical effect of utilizing the disclosed subject mattermay be to enable automatic modification of a GUI representation, withoutuser intervention. Using machine learning techniques, the disclosedsubject matter may define the representation set automatically.Additionally or alternatively, even when the representation set may failand provide an indefinite result, the disclosed subject matter mayprovide for an automated manner of correcting the representation set toprovide a definite result and acquire the target GUI element correctly.

Yet another technical effect of utilizing the disclosed subject matteris to consume computational resources in the agent, for robustnesspurposes. Instead of utilizing a single representation to acquire theGUI element, a plurality of alternative and independent representationsare used, and therefore consume computational resources. In someexemplary embodiments, the number of alternative representations may bedetermined as a balance between maximal robustness (e.g., any potentialrepresentation) and between best performance (e.g., one representation).

The disclosed subject matter may provide for one or more technicalimprovements over any pre-existing technique and any technique that haspreviously become routine or conventional in the art. Additionaltechnical problem, solution and effects may be apparent to a person ofordinary skill in the art in view of the present disclosure.

Referring now to FIG. 1 showing an environment, in accordance with someexemplary embodiments of the disclosed subject matter.

Environment 100 may comprise a plurality of Client Devices 110. ClientDevices 110 may by Personal Computers (PCs), tablets, smartphones, orthe like. Client Devices 110 may be connected to a Computerized Network105, such as a Wide Area Network (WAN), a Local Area Network (LAN), awireless network, the Internet, an intranet, or the like.

Environment 100 may comprise Server 120, which may be connected toComputerized Network 105. Additionally or alternatively, AdministratorComputer 130, operated by Administrator 140 or another user withsuitable credentials and permissions, may be connected to ComputerizedNetwork 105. In some exemplary embodiments, Server 120 and AdministratorComputer 130 may be connected to each other directly, indirectly, suchas via Computerized Network 105, or the like. In some exemplaryembodiments, Server 120 and Administrator Computer 130 may beimplemented by the same physical device.

In some exemplary embodiments, an Agent 115 may be running on a ClientDevice 110. In some exemplary embodiments, each Client Device 110 mayexecute a corresponding Agent 115. Additionally or alternatively, only aportion of Client Devices 110 may execute an Agent 115. Agent 115 may bea program product executable by a computer, such as, withoutlimitations, a script, a software, a browser extension, a mobileapplication, a web application, a Software Development Kit (SDK), ashared library, a Dynamic Link Library (DLL), or the like.

Client Device 110 may be configured to execute a program having a GUI.The program may be, for example, a web page, a web application, abrowser extension, a mobile application, desktop application, or thelike. The program may display to an end-user one or more screens,constituting the GUI, which comprise GUI elements.

In some exemplary embodiments, Agent 115 may be configured to acquire aGUI element appearing in the GUI. For example, Agent 115 may acquire aspecific interactive button, a text element, a tool tip, or the like.GUI element acquisition may be performed in order to for Agent 115 toimplement a desired functionality. In some exemplary embodiments, Agent115 may track user activity with respect to the program, and inparticular, monitor usage statistics relating to GUI element.Additionally or alternatively, Agent 115 may implement GUI-basedautomation, automatically interacting with the GUI on behalf of theend-user, mimicking user-input. Additionally or alternatively, Agent 115may implement GUI manipulation, such as modifying the GUI, removingexisting GUI elements, adding new GUI elements, or the like.Additionally or alternatively, Agent 115 may implement a walkthrough, inwhich elements of the walkthrough may be displayed in a locationadjacent GUI elements, provide call for actions for the end user,relating to GUI elements displayed in the GUI, or the like.

In some exemplary embodiments, Agent 115 may be configured to obtain arepresentation set of the GUI element. Agent 115 may apply therepresentation set on the GUI, so as to acquire the GUI element in theGUI. The representation set may comprise a plurality of alternativerepresentations, each of which may provide a manner of identifying theGUI element in the GUI. In order to apply the representation set, Agent115 may be configured to apply each of the alternative representationsin a separate manner.

In some exemplary embodiments, applying the representation set may failto identify any GUI element in the GUI. Additionally or alternatively,applying the representation set may provide indefinite results due toidentifying a first GUI element using a first alternative representationand identifying a second GUI element using a second alternativerepresentation, wherein the first GUI element and the second GUI elementare different. Additionally or alternatively, applying therepresentation set may provide indefinite results due to identifying twoor more GUI elements using an alternative representation.

As an example, it may be desired to acquire a “new” button appearing inthe GUI and relating to composing a new email, which appears in theemail container. In some exemplary embodiments, one representation thatmay be used to acquire the GUI element, may be to search for an elementhaving the properties “Type=Button” and “Label=new”. Another alternativerepresentation may be an element having the properties “Type=Button” and“Parent=Email Container”. The representation set may comprise bothalternative representations, each of which may be sufficient to identifythe GUI element. However, in some exemplary embodiments, the GUI maycomprise a plurality of containers, such as email container, contactcontainer, event container, or the like, each of which may comprise a“new” button. As a result, the first alternative representation mayprovide three different GUI elements when applied on the GUI, as theremay be three different elements that match the properties definedthereby. The second alternative representation may still provide aunique result—the GUI element. Such discrepancy may be considered as therepresentation set providing an indefinite result.

Upon determining that applying the representation set provides anindefinite result, Agent 115 may be configured to communicate theinformation to Server 120. Server 120 may be configured to obtain suchinformation and provide a modified representation set.

In some exemplary embodiments, applying an alternative representationmay not provide a result, may provide an empty result, or the like. Itis noted that such alternative representation may be correct. As anexample, a feature may be available in one Operating System (OS) and notavailable in another. A first version of the GUI may have been identicalfor two different OSs. The feature may be represented using a button.The first version of the GUI may comprise the button in all OSs. Asecond version of the GUI may be OS specific and may include the buttonfor one OS and not include the button for another OS. Applying thealternative representation on the first version of the GUI may providethe button regardless from which OS the GUI was obtained. Applying thealternative representation on the second version of the GUI may providethe button when the GUI is obtained from a first OS. Additionally oralternatively, applying the alternative representation on the GUI thatwas obtained from a second OS may not provide a result, as the GUIelement is indeed absent from that GUI.

In some exemplary embodiments, Server 120 may retain GUIs. In someexemplary embodiments, Agents 115 may be configured to transmit toServer 120 GUIs encountered thereby. In some exemplary embodiments, eachencountered GUI may be transmitted. Additionally or alternatively, onlyGUIs where the representation set provides an indefinite result may betransmitted. In some exemplary embodiments, a GUI that is retained maybe a digital representation of the displayed GUI, such as an Hyper TextMarkup Language (HTML) document, a screen capture image, eXtensibleMarkup Language (XML) document representing the elements of the GUI andtheir properties, a Document Object Module (DOM) file, ANDROID™ layout,or any other digital representation of the GUI.

In some exemplary embodiments, Server 120 may be configured to define amodified representation set. In some exemplary embodiments, the modifiedrepresentation set may be defined automatically, manually, or the like.The modified representation set may be defined using a different set ofalternative representations. In some exemplary embodiments, the modifiedrepresentation set may be determined in view of information obtainedfrom one Agent 115 or a plurality of Agents 115. In some exemplaryembodiments, the modified representation set may be transmitted byServer 120 to Agents 115. In some exemplary embodiments, Server 120 maybroadcast the modified representation set upon its generation.Additionally or alternatively, Server 120 may transmit to each Agent 115the modified representation set, in response to a query from that agent.

Environment 100 may comprise Administrator Computer 130 and User 140that may be an operator, a supervisor, an administrator, or the like. Insome exemplary embodiments, in response to determining that applying therepresentation set provides an indefinite result, the information may becommunicated to Administrator Computer 130. Administrator Computer 130may be configured to display the information to Administrator 140.Additionally or alternatively, Administrator Computer 130 may beconfigured to display to Administrator 140 the GUI, as captured by theagent in which the representation set provided the indefinite result.User 140 may make a selection of the GUI element in the displayed GUI.In response to obtaining from Administrator 140 a selection of the GUIelement in the GUI, the modified representation set may be determined.In some exemplary embodiments, the modified representation set maycomprise alternative representations, where each of the alternativerepresentations uniquely identify the selected GUI element. In someexemplary embodiments, the alternative representations may be selectedfrom a set of potential alternative representations, such as retained byServer 120 or generated thereby.

Additionally or alternatively, Administrator Computer 130 may beconfigured to display to Administrator 140, the original GUI, based onwhich the representation set was determined, with a visual indication ofthe GUI element, as identified by the representation set. The displaymay further include the current GUI, in which the representation setprovided an indefinite result. In some exemplary embodiments, thedisplay may show the original GUI and the current GUI in a side-by-sidemanner, enabling Administrator 140 to visually view both GUIs at thesame time, compare them, and identify the correlation between theelements thereof. Additionally or alternatively, the display maycomprise a visual indication of an estimation of the GUI element. Theestimation may be determined, for example, based on a majority of thealternative representations providing a potential candidate.Additionally or alternatively, each alternative representation may beassociated with a weight and based on the weights, a score for eachcandidate may be computed, enabling a selection of atop-scoredcandidate. Additionally or alternatively, a prediction model may beutilized to determine a predicted selection based on the candidates. Insome exemplary embodiments, the display may by centered on the estimatedGUI element. Administrator 140 may verify whether the estimation of GUIelement is correct, or alternatively indicate it is incorrect. In someexemplary embodiments, in response to an indication of an incorrectestimation, a next-best estimation may be displayed, and so forth.Additionally or alternatively, upon displaying the next estimatedcandidate, the display may be re-centered on the next estimatedcandidate. Additionally or alternatively, Administrator 140 may manuallyselect the GUI element in an explicit manner. In some exemplaryembodiments, user input may be used by a machine learning algorithm,such as supervised learning, semi-supervised learning, non-supervisedlearning, or the like, to provide a predictive model capable ofreplacing a human user by an automated classifier.

In some exemplary embodiments, Agent 115 may be configured to obtain themodified representation set and utilize it as the representation set foracquiring the GUI element. In some exemplary embodiments, Agent 115 mayapply the modified representation sets on the current GUI or onadditional GUIs.

In some exemplary embodiments, attempts to apply the representation setmay be retained and used by a machine learning algorithm, so as to allowimproved selection of alternative representations to be used as part ofthe representation set. The improved selection may take into accountproperties of the GUI element, the properties of the alternativerepresentation being examined and use a predictive model to estimaterobustness of the examined alternative representation over time.Additionally or alternatively, the improved selection may take intoaccount other alternative representations that uniquely identify the GUIelement as part of the features used to estimate the robustness of thealternative representation. Additionally or alternatively, the improvedselection may take into account other alternative representations thatprovided indefinite results (of specific type, or in general),alternative representations that were exhibited to be non-robust (e.g.,initially provided a unique identification and later on provided anon-unique identification), or the like. Additionally or alternatively,the improved selection may take into account GUI context, properties ofthe GUI, or the like.

In some exemplary embodiments, a GUI may be associated with a GUIcontext. The GUI context may be properties relating to the GUI, to theuser using the GUI, to the device being used to execute the GUI, to theconnectivity status of the device, or the like. As an example, GUIcontext may comprise an Operating System (OS) of the device being used.As another example, the GUI context may comprise a locale of the device,such as language settings, regional location, time zones, or the like.As yet another example, the GUI context may comprise an identifier ofthe GUI, such as a Unified Resource Identifier (URI), a screen name, orthe like. As yet another example, the GUI context may compriseproperties of the logged-in user, such as permissions, settings, roles,or the like. As yet another example, the GUI context may comprisehardware properties of the device, such as screen size, orientation,resolution, network connectivity, or the like. In some exemplaryembodiments, a first Agent 115, running on a first Client Device 110,may uniquely identify a GUI element, while a second Agent 115, runningon a second Client Device 110, may provide indefinite result for thesame GUI, due to a difference in the GUI context.

Referring now to FIG. 2 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 200, a selection of GUI element is obtained. In some exemplaryembodiments, a user may select a GUI element from a GUI, also referredto as a screen, to be used as part of an automation process, of awalkthrough, or the like. Additionally or alternatively, the selectionmay be performed automatically, relating to elements of interest, suchas elements for which performance measurements may be tracked. In someexemplary embodiments, the selected GUI element may be a GUI elementthat may be acquired in the future, such as when the program having theGUI is invoked. In some exemplary embodiments, the selection may berepresented in a digital form, such as using a pointer to the selectedGUI element in an application, an identifier of the selected GUI elementin a web page, or the like. The GUI may part of a program, such as amobile application, a web application, a desktop software, or the like.Additionally or alternatively, the GUI may be a screen previouslypresented by a program and retained in a database.

As an example, in some situations, when rendered, the GUI of a programmay be Google™ home page. The GUI element may be the search area inwhich users may type a text. The search area may be obtained bysearching for “div id=searchBox”. As another example, the GUI may be ascreen in a Google™ Search mobile application, instead of a webapplication. The screen may comprise a GUI element such as a search box,which may be identified as “androidid=“searchBox””.

On Step 210, a representation set of the GUI element is determined. Therepresentation set may comprise a plurality of alternativerepresentations, each of which may provide a manner of identifying theGUI element in the GUI of a program.

Referring again to the above example, the search box may havealternative representations that can be used to identify the selectedGUI element, such as “div id=searchBox”, “GUI element with the text‘Search Google or type a URL’”, “the 7^(th) div in the page”, or thelike.

On Step 220, a GUI from a historical database may be obtained. In someexemplary embodiments, the historical database may retain GUIs, such asscreens previously encountered by computerized agents, such as webcrawlers, Agents 115 of FIG. 1, screens manually saved or archived byusers, or the like. In some exemplary embodiments, the GUI obtained fromthe historical database may be a different version of the GUI from whichGUI element was selected. Additionally or alternatively, the historicalGUI may comprise the GUI element and as a result, the GUI element may beacquired therein. In some exemplary embodiments, the historical GUI mayhave the same GUI context as the GUI from which the GUI element wasselected or have a shared common property therebetween. As an example,if the GUI element was selected from a webpage having a Unified ResourceLocator (URL), historical GUIs of webpages having the same URL may beobtained on Step 220. As another example, the historical GUIs may bescreens having the same identifier as the GUI from which the GUI elementwas selected.

On Step 230, the representation set may be applied on the GUI obtainedon Step 220. Applying the representation set may comprise applying eachof the alternative representations comprised by the representation set.The results of said applying each of the alternative representations maybe retained and compared to one another.

In some exemplary embodiments, some alternative representations may beassociated with a GUI context. As an example, the GUI context maycomprise a property describing the OS from which the GUI was obtained.The property may be “OS=ANDROID™”, “OS=WINDOWS”, or the like. Saidapplying the representation set may comprise selecting a subset of thealternative representations based on the GUI context and applying onlythe subset, for the purpose of uniquely identifying the GUI element.Referring again to the above example, the homepage of Google™ may bedisplayed differently for ANDROID™ smartphones. As an example, thesearch box may be placed in the 8^(th) div element and the id may be“searchBox_android”. Applying the alternative representation “divid=searchBox” on the updated web page, may not provide any GUI elementwhen the GUI context comprises OS=ANDROID. Additionally oralternatively, the alternative representation “the 7^(th) div in thepage” may provide a different GUI element than the search box, andshould not be used when the GUI context comprises OS=ANDROID. In somecases, for OS=ANDROID, different alternative representations may beavailable, such as “the 8^(th) div in the page”, “divid=searchBox_android”, or the like. Furthermore, the alternativerepresentation “GUI element with the text ‘Search Google or type a URL’”may be applicable to all operating systems and indifferent to theproperty of OS type in the GUI context. As a result, this alternativerepresentation may be applied for both OS=ANDROID GUI context as well asOS=WINDOWS GUI Context.

On Step 240, it may be determined whether applying the representationset in Step 230 provided an indefinite result. In some exemplaryembodiments, the representation set may comprise two differentalternative representations: a first and a second alternativerepresentation. Each of the different alternative representation mayprovide a unique identification of a different GUI element. Additionallyor alternatively, an indefinite result may be due to a singlealternative representation identifying two different GUI elements, notproviding any GUI element, or the like.

In some exemplary embodiments, in case the representation set provided adefinite result using the GUI obtained in Step 220, additional versionsof the GUI may be obtained and the representation set may be appliedthereon on Steps 220-240.

In case that the representation set provided an indefinite result (240),Step 250 may be performed. On Step 250, the representation set may bemodified. In some exemplary embodiments, the modification of therepresentation set may be performed to improve the representation set.Additionally or alternatively, the modification may be aimed at creatinga more robust representation that would provide definite results indifferent versions of the GUI.

On Step 260, an alternative representation may be removed from therepresentation set. In some exemplary embodiments, a representation setmay comprise a plurality of alternative representations. Applying therepresentation set may provide an indefinite result due a firstalternative representation. The first alternative representation may beremoved from the representation set. The modified representation set maycomprise the alternative representations without the first alternativerepresentation. Re-applying the modified representation set in a GUI ofthe program may provide a definite result.

Referring again to the example comprising Google™ home page, arepresentation set comprising the alternative representations “divid=searchBox”, “GUI element with the text “Search Google or type aURL””, “the 10th div in the page”; may provide an indefinite result.Removing the alternative representation “the 10th div in the page” mayyield a modified representation set. Re-applying the modifiedrepresentation set in a GUI of the program provides a definite result.The result may be the search box in the web page.

On Step 270, another alternative representation may be added to themodified representation set. The another alternative representation maybe a representation that, when applied on the screen of Step 220, wouldprovide a definite result, consistent with the other alternativerepresentations of the modified representation set. In some exemplaryembodiments, the another alternative representation may be added inresponse to determining that the number of alternative representationsin the modified representation set is below a minimal threshold number,such as below five, below ten, or the like.

In some exemplary embodiments, an original screen may be stored. Theoriginal screen may be a screen comprising the original GUI element. Theoriginal screen may be the screen using which the representation set forthe GUI element was originally determined. In some exemplaryembodiments, the another alternative representation may be added only ifthe another alternative representation also identifies the GUI elementin the original screen. Such restriction may be useful for ensuring thatthe newly added alternative representation that is added to therepresentation set, would not lead to an indefinite result, should themodified representation set be applied on the original screen. In someexemplary embodiments, the original screen may be an historic screen,that is no longer relevant. In some cases, the original screen may bereplaced by another historic screen, such as after a major release,modifications caused by an operation system update, or the like. Thescreen that replaces the original screen may be used instead of theoriginal screen to ensure robustness of the another alternativerepresentation.

In some exemplary embodiments, Steps 220-250 may be repeated, using themodified representation set, determined on Step 250. The modifiedrepresentation set may be applied on other versions of the screen, asretained in the history database. In some exemplary embodiments, screensnot yet analyzed in Steps 220-240 may be processed. Additionally oralternatively, screens that were analyzed using the previousrepresentation set, which was modified due to a determination ofindefinite results on Step 240, may also be re-processed, such as toensure that the modified representation set yield definite results withrespect to them. In some cases, the screens that are re-processed may bere-processed after processing screens that were not previouslyprocessed.

Referring now to FIG. 3 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 302, a server may determine a representation set. Thedetermination may be similar to Step 210 of FIG. 2. In some exemplaryembodiments, the server may determine the representation set to be usedfor acquiring a GUI element, such as selected by a user, for preforming,in the future, an action by a client device.

On Step 306, the server may broadcast the representation set to aplurality of client devices. in some exemplary embodiments, the servermay transmit the representation set to each of the client agents, suchas Agents 115 of FIG. 1. Additionally or alternatively, the server maystore the representation set on a data storage, accessible to the clientdevices, thereby allowing the client devices to retrieve therepresentation set, in a pull model.

On Step 310, a client device may obtain the representation set. Therepresentation set may be obtained from the server, and as a result ofthe server broadcasting the representation on Step 306.

On Step 314, a client device may obtain a screen or another GUI. Thescreen may be obtained for the purpose of performing an action, such asbut not limited to, tracking usage of the screen, presenting a digitalwalkthrough, performing an automated process, performing automatedtesting, modifying the visual representation of the screen, presentinganother display layer over the screen, augmenting the functionality ofthe screen, performing analytics, or the like. In some exemplaryembodiments, the action may be performed with respect to a GUI elementof the screen. In some exemplary embodiments, the screen may be obtainedby an agent such as Agent 115 of FIG. 1.

On Step 318, the representation set may be applied on the screen. Theapplication of the representation set may be performed similarly to Step230 of FIG. 2. In some exemplary embodiments, the representation set maybe applied so as to acquire, by the client device, the GUI element, toallow performing the action.

On Step 320, it may be determined whether the application of therepresentation set yielded an indefinite result. In case that anindefinite result is provided, the representation set had failed touniquely identify the GUI element and it may be desired to modify therepresentation set to improve its robustness. Steps 330-350 may beperformed. In the other cases, if the GUI element was uniquelyidentified using the representation set, Step 360 may be performed.

On Step 330, the information may be communicated to the server. Theinformation may comprise the screen, the GUI context, the representationset that was applied, the results of applying the representation set, orthe like. In some exemplary embodiments, the information may be providedto the server, so as to improve the representation set. Additionally oralternatively, the server may retain the information to be used by amachine learning algorithm, for quality assurance, or the like.

On Step 340, the server may determine a modified representation set. Insome exemplary embodiments, the server may be configured to modify therepresentation set in order to determine the modified representationset. In some exemplary embodiments, the representation set may bemodified in a similar manner to that of Step 250 of FIG. 2. In someexemplary embodiments, the modification may be based on user input,performed automatically using machine learning, using combinationthereof, or the like.

On Step 350, client devices may obtain the modified representation set.In some exemplary embodiments, the server may be configured to transmitthe modified representation set to the plurality of client devices ofStep 306. In some exemplary embodiments, the client devices may beupdated with the modified representation set, so as to enable each ofthe client devices, when obtaining a screen and attempting to acquirethe GUI element (e.g., Steps 314-318), to use the modifiedrepresentation set, instead of the original representation set.

On Step 360, an action may be taken by the client device regarding theGUI element that was acquired. In some exemplary embodiments, anacquired GUI element may be GUI element that is uniquely identified byan application of a representation set. Additionally or alternatively,in case the representation set yielded an indefinite result, anestimated acquired element may be selected and Step 360 may be performedwith respect to the estimated acquired element, such as based on aselection criteria, e.g., majority vote, weighted selection, or thelike. It is noted that the action may be taken on the client device withrespect to an estimated acquired element, which may or may not be thecorrect GUI element. The action may be taken irrespective to themodified representation set, which may not yet be available at the timethe acquisition is performed. For example, the modified representationset may be validated by a human user, who may validate it within areasonable timeframe, such as, an hour, 24 hours, 72 hours, 2 weeks, orthe like. The client device may require an immediate acquisition, on thefly, of the GUI element, for its purposes. Such acquisition may beperformed using the estimation, which may be available immediately.

In some exemplary embodiments, Steps 318-360 may be repeated as long asthe client agent is running. In some exemplary embodiments, theinformation collected by the client agent may be communicated to theserver. The information may be used in a machine learning algorithmconfigured to optimize the determination of representation sets overtime.

Referring now to FIG. 4 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 400, a representation set may be determined. The determinationmay be performed by a server, such as Server 120 of FIG. 1. In someexemplary embodiments, the determination may be based on history ofpreviously obtained screens. In some exemplary embodiments, determininga representation set may comprise Step 410 and Step 420.

In some exemplary embodiments, a crawler bot may be configured to crawlover web sites, retrieving the web documents of each web page. Theretrieved web documents may be retained in a historic database.Additionally or alternatively, plurality of computerized agents may bedeployed in client devices, and obtain each screen viewed by the client,generally or with regard to a specific set of tracked programs. Eachsuch screen may be transmitted to the server and retained in thehistoric database.

On Step 410, a set of alternative representations may be determined. Insome exemplary embodiments, an alternative representation may compriseone or more attributes of a GUI element. Additionally or alternatively,an alternative representation may comprise a relative path to a GUIelement in a hierarchy of elements. Additionally or alternatively, analternative representations may comprise an absolute path to the GUIelement in hierarchy of a GUI element. In some exemplary embodiments,all possible combinations of attributes representing the element, pathsleading to the elements, attributes representing intermediate elementsand paths between the intermediate elements and the GUI element may bedetermined. In some exemplary embodiments, the number of generatedalterative representations may be capped by a maximal number, such as1,000 representations.

On Step 420, a subset of alternative representations is selected. Thesubset may be selected from the set of alternative representations thatwas determined on Step 410. In some exemplary embodiments, the selectionmay be of a subset with a limited number of alternative representations.In some exemplary embodiments, there may be a maximal number ofalternative representations, such as about 5, about 10, about 20. Therepresentation set may include exactly or no more than the maximalnumber of alternative representations. In some exemplary embodiments,the maximal number of alternative representations may reflect a balancebetween robustness and performance, in view of each agent needing toapply each alternative representation in production, potentially wastingcomputational resources. In some exemplary embodiments, the maximalnumber of alternative representations may provide a limitation on thenumber of alternative representations applicable per context of the GUI.

In some exemplary embodiments, the selection of the alternativerepresentations to the subset may be based on a selection criteria, suchas an estimated robustness score of each alternative representation.Additionally or alternatively, the selection may be performed so as tomaximize an estimated robustness score of the representation set, suchas in view of different alternative representation accounting fordifferent aspects of the GUI element. For example, some selectedrepresentations may be based on the textual attributes of the element,where others may be based on hierarchical properties of the element andyet others may be based on visual attributes of the element. It may beconsidered unlikely that all different aspects would be modified at thesame time, and therefore the representation may continue to be usefulfor acquiring the GUI element.

On Step 430, an indication may be received. The indication may beindication of an application of the representation set yielding anindefinite result. In some exemplary embodiments, the indication mayhave been sent by an agent, such as Agent 115 of FIG. 1. The indicationmay comprise the representation set that was applied, the GUI context,the GUI on which the representation set was applied (e.g., a screen), orthe like.

On Step 440, in response to receiving the indication of Step 430, amodified representation set may be determined. The determinedrepresentation set may comprise a different subset of the previouslygenerated set of alternative representations, determined in Step 410. OnStep 450, the different subset may be selected. The selection may beperformed based on the robustness score, in view of which alternativerepresentation has failed to provide a correct result. In some exemplaryembodiments, instead of retaining the set of alternative representationsof Step 410, the set may be re-computed and a selection may be madetherefrom.

Referring now to FIG. 5A-5E, showing illustrations of screens, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

FIG. 5A may illustrate a web page of a sales portal. The GUI of the webpage is shown as Screen 500 a. A user by the name of John Doe may belogged in (503 a). Screen 500 a comprises Menu 501 a. Menu 501 acomprises categories such as Home 511 a, Accounts 502 a, Contacts 505 a,or the like. Currently, Screen 500 a visually indicates that Home 511 ais selected and that Pane 520 a currently shows the “home” sub-page.

When the menu item Accounts 502 a is pressed, a new sub-page, “Accounts”may be loaded to Pane 520 a. Additionally or alternatively, the newsub-page may be loaded by loading a different web page, by replacing thecontent of Pane 520 a, or the like.

Menu item Accounts 502 a may be acquired by applying a representationset, in accordance with the disclosed subject matter. The representationset may be based on several alternative attributes of Accounts 502 a. Asan example, the representation set may comprise two alternativerepresentations. The first alternative representation may be being “the5^(th) menu item under the menu”. The second alternative representationmay be “menu item AND title=Accounts”.

FIG. 5B may illustrate another web page of the same sales portal as FIG.5A. FIG. 5B shows Screen 500 b. However, in this embodiment of the GUI,the 5^(th) element in Menu 501 b is Contacts 505 b, and not Accounts 502b. The difference between the order of menu items between Screens 500 aand 500 b may be the result of rendering in a different platform,programmatic change of the web page, or the like. In some exemplaryembodiments, the difference may be a result of rendering the same sourcecode file using different configurations.

As can be appreciated, if the same representation set described withrelation to FIG. 5A would be applied on FIG. 5B, the result would beindefinite. The first alternative representation would point to Contacts505 b, while the second alternative representation would point to thecorrect target GUI element, Accounts 502 b. As can be appreciated, thefirst alternative representation is not robust and does not guaranteeacquisition of the correct element in different embodiments of the sameGUI. in some exemplary embodiments, the representation set may compriseadditional alternative representations and a majority vote may be usedto select the correct element in between the potential elements foracquisition, 502 b and 505 b.

In some exemplary embodiments, in case the representation set providedan indefinite result when applied on Screen 500 b, an administrator maybe shown Screen 500 a, with a highlighting of the target GUI element,Accounts 502 a, side-by-side with Screen 500 b, with a highlighting ofthe potentially acquired elements. In some exemplary embodiments, a mostlikely correct element out of a plurality of potentially identifiedelements may be shown to the user for selection. For example, if therepresentation set identified both elements 502 b and 505 b as potentialelement for acquisition based on the alternative representations, one ofsuch elements may be highlighted, and the user may confirm or reject theselection. Such a representation may provide with the user a simplegraphical manner to visually determine whether the acquired element inScreen 500 b is indeed the element whose acquisition was defined basedon Screen 500 a. Additionally or alternatively, in case no potentialcandidate for acquisition is identified, the user may select the elementfor acquisition from Screen 500 b, while viewing the element as shown inScreen 500 a.

In some exemplary embodiments, alternative representations may beassociated with different GUI contexts, and applied only to screensmatching their respective GUI contexts. In some exemplary embodiments, aGUI context may comprise the configuration file. The configuration filemay indicate whether the screen being displayed is Screen 500 a orScreen 500 b. In such a case, it may be determined that an alternativerepresentation that is based on the 5^(th) position in the menu (“the5^(th) menu item under the menu”) can be used in the configurationindicating Screen 500 a, whereas a different alternative representation,such as “the 4^(th) menu item under the menu”, may be applicable to theconfiguration indicating Screen 500 b.

In some exemplary embodiments, the GUI context may be determinedautomatically by a context resolution algorithm. The context resolutionalgorithm may collect information from the device on which the GUI isdisplayed such as the hardware of the device, the OS of the device, orthe like. Additionally or alternatively, the context resolutionalgorithm may obtain data from files that are associated with theprogram displaying the GUI such as cookies files, configuration files,or the like. Additionally or alternatively, the context resolutionalgorithm may obtain data from a back end server configured to serve theprogram displaying the GUI.

FIG. 5C may illustrate a Screen 500 c, showing a “contacts” web page,where Contacts 505 c is selected. Screen 500 c is a screen of a webbrowser, comprising also a URL Field 502 c, indicating the URL being“ga.com/contacts”. In some exemplary embodiments, Screen 500 c may beloaded after a user pressed Contacts 505 a in Screen 500 a of FIG. 5A.Screen 500 c comprises New Button 506 c. New Button 506 c has the label“New”. Pressing New Button 506 c may, for example, create a new contact.

FIG. 5D may illustrate a Screen 500 d, showing an “opportunity” webpage, where Opportunities 512 d is selected. Screen 500 d comprises URLField 502 d, indicating the URL being “ga.com/opportunities”. The webpage may be loaded after a user pressing Opportunity 512 a of FIG. 5A.Screen 500 d also comprises New Button 506 d. Similarly to New Button506 c, New Button 506 d is also labeled simply as “New”. However, theirfunctionality is different, as clicking on New Button 506 d may create anew opportunity rather than a new contact.

A representation set may comprise an alternative representation such as“Type=Button” and “Text=New”. Applying that alternative representationon Screen 500 c may identify New Button 506 c, while applying thatalternative representation on a Screen 500 d may identify New Button 506d. In spite of the label “New” for a button being a unique identifier ineach such screen, such representation is not a robust representation ofthe relevant button having the desired functionality within the programitself. Hence, the same representation might be utilized as part of therepresentation set for representing the new contacts button, but wouldresult in an indefinite result when applied on Screen 500 d.

In some exemplary embodiments, the alternative representation“Type=Button and Text=New” may be applicable only in the GUI context ofthe URL being “ga.com/contacts”, having a suffix of “/contacts”, or thelike. In that case, applying the representation set on Screen 500 c mayidentify New Button 506 c. Applying that representation set on Screen500 d may provide an empty result, in view of the alternativerepresentation not being applied due to the different GUI context.

FIG. 5E illustrates Screen 500 e similar to that shown in FIG. 5A.Augmented functionality to Screen 500 a may provide an additionalelement not present in the original GUI (Screen 500 a). Additionally oralternatively, a display layer may add elements to the GUI. As anexample, a tooltip element may be introduced into Screen 500 a andpresented in 500 e. Container 521 e may be acquired and modified tointroduce a Tooltip Element 510 e, which is displayed when a “mousehover” event is identified with respect to Container 521 e. A robust andautomatic acquisition of Container 521 e may therefore be desired.

Referring now to FIG. 6A, showing an apparatus in accordance with someexemplary embodiments of the disclosed subject matter.

In some exemplary embodiments, Apparatus 600 may be configured toprovide backend services to client devices. Additionally oralternatively, Apparatus 600 may be configured to determiningrepresentation sets of one or more GUI elements. Additionally oralternatively, Apparatus 600 may be able to transmit, broadcast, or thelike a representation set to a client device, to a plurality of clientdevices, or the like.

In some exemplary embodiments, Apparatus 600 may comprise one or moreProcessor(s) 602. Processor 602 may be a Central Processing Unit (CPU),a Graphical Processing Unit (GPU), a microprocessor, an electroniccircuit, an Integrated Circuit (IC) or the like. Processor 602 may beutilized to perform computations required by Apparatus 600 or any of itsubcomponents.

In some exemplary embodiments of the disclosed subject matter, Apparatus600 may comprise an Input/Output (I/O) Module 604. I/O Module 604 may beutilized to provide an output to and receive input from a user, such as,for example obtaining selection of GUI elements, obtaining aconfirmation regarding applying a representation set, or the like. As anexample, applying a representation set may provide an indefinite result.A user may observe two or more different results and may provide aselection of a GUI element.

In some exemplary embodiments, Apparatus 600 may comprise Memory Unit606. Memory Unit 606 may be a hard disk drive, a Flash disk, a RandomAccess Memory (RAM), a memory chip, or the like. In some exemplaryembodiments, Memory Unit 606 may retain program code operative to causeProcessor 602 to perform acts associated with any of the subcomponentsof Apparatus 600. In some exemplary embodiments, Memory Unit 606 mayretain representation sets to be distributed to agents, alternativerepresentations for each element, historic database of screens (e.g.,retained in Historic Screens Database 660), or the like.

Memory Unit 606 may comprise one or more components as detailed below,implemented as executables, libraries, static libraries, functions, orany other executable components.

Memory Unit 606 may comprise a Representation Determinator 610.Representation Determinator 610 may be configured to determine arepresentation set for a GUI element in a GUI of a program. In someexemplary embodiments, Representation Determinator 610 may be configuredto generate a set of alternative representations of the GUI element tobe used in a representation set. In some exemplary embodiments,Representation Determinator 610 may generate a set of alterativerepresentations, a subset of which may be selected and used in therepresentation set. In some exemplary embodiments, RepresentationDeterminator 610 may obtain screens from Historic Screens Database 660to apply the set of alternative representations thereon. RepresentationDeterminator 610 may apply each of the alternative representationscomprising the subset in order to build a representation set. Byapplying the alternative representations on different screens that wereencountered in the past, it may be estimated which of the alternativerepresentations is robust and would return the correct result over time.Using such information, the representation set may be built, byincluding alternative representations that have proved, using screensfrom Historic Screens Database 660, to provide correct and definiteresults.

Representation Determinator 610 may be configured to store the set ofalternative representations that provided correct results. In someexemplary embodiments, the alternative representations for each GUIelement may be retained in Alternative Representations Database 680.Additionally or alternatively, the alternative representations may beretained together with accuracy metrics, indications of screens in whichthe representation did not function properly, such as GUI contextthereof, indications of screens in which the presentation did functionproperly, or the like.

In some exemplary embodiments, the Representation Determinator 610 maybe configured to determine the representation set to be obtained byclient agents, enabling such agents to acquire the GUI element.

In some exemplary embodiments, a maximum number of allowed alternativerepresentations per representation set may be provided to RepresentationDeterminator 610. As the number of potential alternative representationsmay be very large, such as millions or even billions of potentialcombinations, providing a supremum to the number of alternativerepresentations in the representation set may yield an improvement inthe resource consumption of the disclosed subject matter. Additionallyor alternatively, a minimum number of alternative representations in therepresentation set may be provided to Representation Determinator 610.Providing an infimum to the number of alternative representations mayensure a confidence level to the acquisition of a GUI element. As anexample, Representation Determinator 610 may generate a representationset comprising five (5) alternative representations. The representationset may provide mixed results when functioning in the field. Forexample, in some cases, acquisition of the GUI element using the fivealternative representations may fail, and in other cases, it maysucceed. As a result, the infimum may be increased. For example, insteadof using five (5) alternative representations, ten (10) alternativerepresentations may be used, yielding a higher confidence level ofacquisition compared to a representation set comprising only half asmany alternative representations. In some exemplary embodiments, apotentially optimal number of alternative representations to be used torepresent a GUI element may be automatically learned based on theattributes of the GUI element, in view of the actual outcomes ofapplying representation sets of similar GUI elements. The potentiallyoptimal number may balance performance issues related to applying manyalternative representations, on one hand, and confidence level relatedto applying few alternative representations.

In some exemplary embodiments, Representation Determinator 610 may beconfigured to determine different representation sets for the same GUIelement in different GUI contexts. The GUI context may be indicative forthe platform from which the screen was obtained, indicative tocharacterizations of the user, or the like. As an example, a GUI contextmay comprise the language in which a web site is presented to the enduser. There may be a button comprising a label “END” for the Englishversion and a label “FIN” for the French version. RepresentationDeterminator 610 may determine two different representation sets, onefor each language. In some exemplary embodiments, all representationsets may be distributed to all agents to be selected thereby on-the-fly.Additionally or alternatively, each agent may provide a set of relevantGUI contexts thereto, and obtain the representation sets that match therelevant GUI contexts.

Additionally or alternatively, Representation Determinator 610 may storethe set of alternative representations to be used by RepresentationLearner 650. In some exemplary embodiments, Representation Determinator610 may store the alternative representations that provided a correctresult. Additionally or alternatively, Representation Determinator 610may store the alternative representations that failed to provide thecorrect result. Such information may be useful for automated learningperformed by Representation Learner 650.

Memory Unit 606 may comprise Representation Transmitter 620.Representation Transmitter 620 may be configured to transmit arepresentation set to a client device, such as Apparatus 601 of FIG. 6B,110 of FIG. 1, or the like. In some exemplary embodiments,Representation Transmitter 620 may be configured to broadcast arepresentation set to a plurality of client devices. Additionally oralternatively, the transmission may be invoked by a request from theclient device. In some exemplary embodiments, the transmittedrepresentation set, whether broadcasted, or specifically addressed tothe client device, may be received thereby and processed by the agentexecuted thereon.

Memory Unit 606 may comprise Indefinite Results Obtainer 630. IndefiniteResults Obtainer 630 may be configured to obtain indefinite results fromclient devices. In some exemplary embodiments, a client device may applya representation set, such as in production, that yields an indefiniteresult. The indefinite result may be reported to Apparatus 600 andobtained by Indefinite Results Obtainer 630. In some exemplaryembodiments, Indefinite Results Obtainer 630 may invoke theRepresentation Updater 640 to modify the representation set in view ofthe indefinite result. In some exemplary embodiments, the updating isperformed immediately after receiving the indefinite result.Additionally or alternatively, the updating may be performed aftermanual input is provided. Additionally or alternatively, the updatingmay be performed periodically.

In some exemplary embodiments, the indefinite result obtained byIndefinite Results Obtainer 630 may be stored in Indefinite ResultsDatabase 670. In some exemplary embodiments, Alternative RepresentationsDatabase 680 may be updated to indicate which alternative representationcaused the indefinite result, in which GUI contexts, in which GUI, orthe like.

Memory Unit 606 may comprise Representation Updater 640. RepresentationUpdater 640 may be configured to update the representation set whichprovided an indefinite result. Representation Updater 640 may beresponsive to an instruction by Indefinite Results Obtainer 630. In someexemplary embodiments, in response to every indefinite result that isobtained, Representation Updater 640 may be configured to modify therepresentation set by removing therefrom one or more alternativerepresentations. Additionally or alternatively, Representation Updater640 may be configured to add new alternative representations to therepresentation set. In some exemplary embodiments, the new alternativerepresentations may be selected from a set of alternativerepresentations generated by Representation Determinator 610, or from asimilar set generated on demand by Representation Updater 640.

In some exemplary embodiments, the indefinite result may include two ormore different candidate elements for acquisition. RepresentationUpdater 640 may display the candidate elements, or portion thereof, toan administrator user, offering the administrator user to select thecorrect element for acquisition. In some exemplary embodiments, thedisplay may show to the administrator user the original screen and theacquired GUI element thereon, as well as the current screen whichresulted in the indefinite result. In some exemplary embodiments, thecandidate elements may be highlighted in the current screen. In someexemplary embodiments, candidates having a likelihood score of beingcorrect above a predetermined threshold may be highlighted. Additionallyor alternatively, a top candidate may be highlighted. Based on theadministrator user selection, Representation Updater 640 may determinewhich alternative representations misperformed. Additionally oralternatively, the user's selection may be retained in databases, suchas 660-680, and utilized for training a machine learning model toautomatically select between alternative representations when indefiniteresults are determined, such as based on attributes of the GUI element,GUI contexts, parameters of the candidate alternative representations,or the like.

Memory Unit 606 may comprise Representation Learner 650. RepresentationLearner 650 may be configured to train a predictive model based onalternative representations, based on representation sets, based on GUIelements, based on GUIs of programs, or the like. In some exemplaryembodiments, the output of Representation Learner 650 may be applied ona currently obtained selection of a GUI element and on alternativerepresentations in order to select the alternative representations to beincluded in the representation set for the GUI element.

In some exemplary embodiments, Representation Learner 650 may implementa “bag of words” model. Each of the alternative representation may beviewed as a word. Representation Learner 650 may be configured to traina predictive model in which the alternative representations are thefeatures. For each GUI element, the output of Representation Learner 650may be a representation set comprising a plurality of alternativerepresentations, each of which may be associated with a score, allowinga weighted majority decision. In some exemplary embodiments, in casethat applying the representation set provided an indefinite result, amajority decision may provide the GUI element. Additionally oralternatively, the weights may be used to select a subset of theplurality of alternative representations to be used as therepresentation set.

Referring now to FIG. 6B, showing an apparatus in accordance with someexemplary embodiments of the disclosed subject matter. In some exemplaryembodiments, Apparatus 601 may a client device, such as Client Device110 of FIG. 1.

In some exemplary embodiments, Apparatus 600 may comprise one or moreProcessor(s) 602, an I/O Module 604, a Memory Unit 606, or the like.

In some exemplary embodiments, Processor 602 may be utilized to performcomputations required by Apparatus 601 or any of it subcomponents.

I/O Module 604 may be utilized to communicate with other computerizeddevices, such as Server 120 of FIG. 1, Apparatus 600 of FIG. 6A, or thelike, to receive a representation set, to notify the server of anindefinite result and provide the relevant information thereto, toprovide usage information and success rates of each representation set,or the like.

Memory Unit 606 may be utilized to retain representation sets to be usedfor acquisition of GUI elements.

Memory Unit 606 may comprise Screen Obtainer 650. Screen Obtainer 650may be configured to obtain a GUI of a program. As an example, a user(not shown) may use Apparatus 601 for her needs, such as for utilizing aweb-based application, a mobile application or the like. In order toprovide services such as walkthroughs, usage analysis, functionalityaugmentation, or the like, Screen Obtainer 650 may obtain the currentscreen in which the user is using. The screen may be obtained bycapturing a digital representation of the GUI, such as obtaining an XMLfile representing the screen, obtaining an HTML file representing thescreen, obtaining a layout file representing the screen, obtaining aplayground object representing the screen, obtaining a DOM file, such asin JavaScript™-based GUIs, obtaining a screen-capture image of thescreen, or the like.

Memory Unit 606 may comprise GUI Element Acquisition Module 655. GUIElement Acquisition Module 655 may be configured to acquire a GUIelement. A GUI element may be acquired automatically, based on anapplication of a representation set on the GUI, obtained by ScreenObtainer 650. The acquired element may be reached, such as to allowaccess to properties thereof or modification of such properties, using apointer, an identifier, a node in a DOM object, or the like. In someexemplary embodiments, GUI Element Acquisition Module 655 may triggerthe execution of other components of Memory Unit 606 such asRepresentation Set Obtainer 660, Indefinite Acquisition Identifier 665,Indefinite Acquisition Resolver 670, or the like.

GUI Element Acquisition Module 655 may obtain a representation setprovided by Representation Set Obtainer 660. Additionally oralternatively, after applying a representation set, GUI ElementAcquisition Module 655 may be configured to invoke IndefiniteAcquisition Identifier 665 in order to determine whether said applyingthe representation set provided an indefinite result.

Memory Unit 606 may comprise Representation Set Obtainer 660.Representation Set Obtainer 660 may be configured to obtainrepresentation sets, such as from a server. In some exemplaryembodiments, Representation Set Obtainer 660 may receive therepresentation set based on a broadcast of the server, by sending aquery to the server in a pull communication model, or the like.Representation Set Obtainer 660 may obtain the representation set inadvance and store the representation set in Memory Unit 606.Additionally or alternatively, Representation Set Obtainer 660 may beobtained the representation set on demand, when a process that requiresacquisition of a GUI element is made available. In some exemplaryembodiments, Representation Set Obtainer 660 may obtain a plurality ofrepresentation sets, each of which useful for acquiring a different GUIelement.

Memory Unit 606 may comprise Indefinite Acquisition Identifier 665.Indefinite Acquisition Identifier 665 may be configured to identify anindefinite result when attempting to acquire a GUI element. In someexemplary embodiments, in response to GUI Element Acquisition Module 655applying the representation set, Indefinite Acquisition Identifier 665may determine whether the outcome is an indefinite result or a definiteresult. Indefinite Acquisition Identifier 665 may identify an indefiniteresult in case that an alternative representation comprising therepresentation set provided more than one result. Additionally oralternatively, Indefinite Acquisition Identifier 665 may identify anindefinite result in case that two different alternative representationscomprised by the representation set provided different results, such asidentified two or more different elements as the element to be acquired,or identified one element to be acquired on the one hand and no elementto be acquired on the other hand.

Memory Unit 606 may comprise Indefinite Acquisition Resolver 670.Indefinite Acquisition Resolver 670 may be configured to resolveindefinite result of a GUI element for the purpose of acquiring anelement albeit the indefinite result. In some exemplary embodiments,Indefinite Acquisition Resolver 670 may be configured to select a GUIelement from a set of potential candidates that were identified by theapplication of the representation set on the GUI. In some exemplaryembodiments, a majority vote or other quorum decision may be made toselect between the potential candidates. Additionally or alternatively,a score may be associated with each candidate, allowing to use aweighted majority decision. In some exemplary embodiments, the score ofthe candidate may be based on a confidence level associated with thealternative representation that provided the candidate, the number ofalternative representations that provided the candidate, success rate ofapplication of the alternative representation on other GUIs, or thelike. In some exemplary embodiments, Indefinite Acquisition Resolver 670may apply a machine learning model to select the correct candidate outof the potential candidates provided by the application of therepresentation set. In some exemplary embodiments, the machine learningmodel may be provided by Representation Learner 650 of FIG. 6A. In someexemplary embodiments, the predictive model determined by RepresentationLearner 650 of FIG. 6A may be provided to Apparatus 601 and applied soas to resolve indefinite results automatically.

Memory Unit 606 may comprise Server Updater 675. Server Updater 675 maybe configured to send updates to a server such as Server 120 of FIG. 1,Apparatus 600 of FIG. 6a , or the like. In some exemplary embodiments,in response to an indefinite result of applying a representation set,Server Updater 675 may communicate the relevant information to theserver, so as to invoke a modification of the representation set. Insome exemplary embodiments, the information is obtained by IndefiniteResults Obtainer 630 of FIG. 6A. Additionally or alternatively,additional information may be transmitted by Server Updater 675 to theserver. For example, usage statistics of application of representationset may be transmitted. As another example, GUIs obtained by ScreenObtainer 650 may be transmitted to the server, and retained in adatabase, such as Historic Screens Database 660.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method comprising: obtaining multiplerepresentations of a Graphical User Interface (GUI) element in a firstpage, wherein the multiple representations comprise alternativerepresentations of the GUI element that enable to identify the GUIelement in the first page, whereby acquisition of the GUI element in thefirst page is enabled using each alternative representationindividually, wherein the multiple representations comprise attributesof the GUI element in the first page; applying the multiplerepresentations of the GUI element over a second page in order toidentify the GUI element in the second page, wherein said applyingcomprises separately applying each representation of the multiplerepresentations over the second page; identifying the GUI element in thesecond page by determining that all of the multiple representations ofthe GUI element identify a same element in the second page, therebyacquiring the GUI element over the first and second pages; andperforming an operation on the second page based on the GUI elementbeing acquired.
 2. The method of claim 1, wherein the multiplerepresentations comprise multiple respective queries that identify theGUI element in the second page by executing the queries over the secondpage.
 3. The method of claim 1, wherein the first and second pagescomprise two version of a webpage.
 4. The method of claim 3, wherein thetwo version of the webpage comprise an early version and a later versionof the webpage.
 5. The method of claim 3, wherein the two versions ofthe webpage comprise two versions of the webpage are provided indifferent languages.
 6. The method of claim 1, wherein the attributes ofthe GUI element comprise at least one of: an identifier of the GUIelement, a text associated with the GUI element, a color of the GUIelement, and a relative path to the GUI element, wherein the relativepath to the GUI element is defined with respect to another GUI elementin the first page.
 7. The method of claim 1 comprising determining themultiple representations by: applying a first set of representationsover multiple historic pages, wherein the multiple historic pagescomprise the different pages, wherein said applying comprises applying afirst representation from the first set of representations over a pageof the multiple historic pages, and applying a second representationfrom the first set of representations over the page, determining thatsaid applying the first set of representations over the page resultswith an indefinite result, wherein the indefinite result comprises afirst resulting GUI element in the page that is obtained based on thefirst representation, and a second resulting GUI element that isobtained based on the second representation, selecting the firstresulting GUI element as corresponding to the GUI element, and modifyingthe first set of representations to a modified set of representations,wherein the modified set of representations excludes the secondrepresentation, wherein applying the modified set of representationsover the page yields a single resulting GUI element, wherein themodified set of representations comprises the multiple representations.8. The method of claim 7, wherein said selecting the first resulting GUIelement is based on a selection of a human.
 9. The method of claim 7,wherein said obtaining the multiple representations of the GUI elementcomprises automatically performing said determining the multiplerepresentations.
 10. The method of claim 7, wherein said determining themultiple representations is performed at a server.
 11. The method ofclaim 1, wherein the operation comprises performing a usage analysis ofthe GUI element over the first and second pages.
 12. The method of claim1, wherein the operation comprises generating a walkthrough over thesecond page.
 13. The method of claim 1, wherein said applying,identifying, and performing is performed by a software agent that isexecuted at an end device.
 14. The method of claim 1, wherein themultiple representations are obtained at an end device from a server.15. The method of claim 1, wherein said applying and identifying areperformed by a software agent coupled to a web browser, wherein the webbrowser being utilized to present the first and second pages.
 16. Themethod of claim 1, wherein said applying and identifying are performedby a software agent coupled to a native mobile application, wherein thenative mobile application being utilized to present the first and secondpages.
 17. The method of claim 1, wherein said applying comprisesquerying a Document Object Model (DOM) of the second page in order toidentify the GUI element in the DOM of the second page.
 18. An apparatuscomprising a processor and a coupled memory, the processor being adaptedto perform: obtaining multiple representations of a Graphical UserInterface (GUI) element in a first page, wherein the multiplerepresentations comprise alternative representations of the GUI elementthat enable to identify the GUI element in the first page, wherebyacquisition of the GUI element in the first page is enabled using eachalternative representation individually, wherein the multiplerepresentations comprise attributes of the GUI element in the firstpage; applying the multiple representations of the GUI element over asecond page in order to identify the GUI element in the second page,wherein said applying comprises separately applying each representationof the multiple representations over the second page; identifying theGUI element in the second page by determining that all of the multiplerepresentations of the GUI element identify a same element in the secondpage, thereby acquiring the GUI element over the first and second pages;and performing an operation on the second page based on the GUI elementbeing acquired.
 19. A system comprising the apparatus of claim 18,wherein the system comprising a processor that is configured todetermine the multiple representations by: applying a first set ofrepresentations over multiple historic pages, wherein the multiplehistoric pages comprise the different pages, wherein said applyingcomprises applying a first representation from the first set ofrepresentations over a page of the multiple historic pages, and applyinga second representation from the first set of representations over thepage, determining that said applying the first set of representationsover the page results with an indefinite result, wherein the indefiniteresult comprises a first resulting GUI element in the page that isobtained based on the first representation, and a second resulting GUIelement that is obtained based on the second representation, selectingthe first resulting GUI element as corresponding to the GUI element, andmodifying the first set of representations to a modified set ofrepresentations, wherein the modified set of representations excludesthe second representation, wherein applying the modified set ofrepresentations over the page yields a single resulting GUI element,wherein the modified set of representations comprises the multiplerepresentations.
 20. The apparatus of claim 18, wherein the operationcomprises at least one of: performing a usage analysis of the GUIelement over the first and second pages; and generating a walkthroughover the second page.
 21. The apparatus of claim 18 comprising asoftware agent that is coupled to a web browser, wherein said applyingand identifying are performed by the software agent, wherein the webbrowser being utilized to present the first and second pages.
 22. Theapparatus of claim 18 comprising a software agent that is coupled to anative mobile application, wherein said applying and identifying areperformed by the software agent, wherein the native mobile applicationbeing utilized to present the first and second pages.
 23. The apparatusof claim 18, wherein said applying comprises querying a Document ObjectModel (DOM) of the second page in order to identify the GUI element inthe DOM of the second page.
 24. A computer program product comprising anon-transitory computer readable storage medium retaining programinstructions, which program instructions when read by a processor, causethe processor to perform: obtaining multiple representations of aGraphical User Interface (GUI) element in a first page, wherein themultiple representations comprise alternative representations of the GUIelement that enable to identify the GUI element in the first page,whereby acquisition of the GUI element in the first page is enabledusing each alternative representation individually, wherein the multiplerepresentations comprise attributes of the GUI element in the firstpage; applying the multiple representations of the GUI element over asecond page in order to identify the GUI element in the second page,wherein said applying comprises separately applying each representationof the multiple representations over the second page; identifying theGUI element in the second page by determining that all of the multiplerepresentations of the GUI element identify a same element in the secondpage, thereby acquiring the GUI element over the first and second pages;and performing an operation on the second page based on the GUI elementbeing acquired.
 25. The computer program product of claim 24, whereinthe program instructions when read by the processor, cause the processorto determine the multiple representations by: applying a first set ofrepresentations over multiple historic pages, wherein the multiplehistoric pages comprise the different pages, wherein said applyingcomprises applying a first representation from the first set ofrepresentations over a page of the multiple historic pages, and applyinga second representation from the first set of representations over thepage, determining that said applying the first set of representationsover the page results with an indefinite result, wherein the indefiniteresult comprises a first resulting GUI element in the page that isobtained based on the first representation, and a second resulting GUIelement that is obtained based on the second representation, selectingthe first resulting GUI element as corresponding to the GUI element, andmodifying the first set of representations to a modified set ofrepresentations, wherein the modified set of representations excludesthe second representation, wherein applying the modified set ofrepresentations over the page yields a single resulting GUI element,wherein the modified set of representations comprises the multiplerepresentations.
 26. The computer program product of claim 24, whereinthe operation comprises at least one of: performing a usage analysis ofthe GUI element over the first and second pages; and generating awalkthrough over the second page.
 27. The computer program product ofclaim 24, wherein at least a portion of the program instructions arecomprised by a software agent that is coupled to a web browser, whereinsaid applying and identifying are performed by the software agent,wherein the web browser being utilized to present the first and secondpages.
 28. The computer program product of claim 24, at least a portionof the program instructions are comprised by a software agent that iscoupled to a native mobile application, wherein said applying andidentifying are performed by the software agent, wherein the nativemobile application being utilized to present the first and second pages.29. The computer program product of claim 24, wherein said applyingcomprises querying a Document Object Model (DOM) of the second page inorder to identify the GUI element in the DOM of the second page.